MULTI-MODAL UTILE DISTINCTIONS Multi-Modal Utile Distinctions

We introduce Multi-Modal Utility Trees (MMU), an algorithm for autonomously learning decision treebased state abstractions in Partially Observable Markov Decision Processes with multi-modal observations. MMU builds the trees using the Kolmogorov-Smirnov statistical test. Additionally, MMU incorporates the ability to perform online tree restructuring, enabling it to build and maintain a compact state approximation. It then uses reinforcement learning over these abstractions to learn a control policy. We empirically evaluate the performance of MMU on the Tsume-Go subtask of Go, and show knowledge transfer from the single step Tsume-Go task to the multi-step Tsume-Go task. In the single step task MMU outperforms the best existing decision tree based methods and performs competitively with a Support Vector Machine method for solving Tsume-Go problems. We are unaware of existing work on multi-step Tsume-Go, but MMU reaches over 40% accuracy on subsequent gameplay steps. A random agent would achieve less than 9%, and thus there is a significant amount of knowledge transfer. The use of multi-modal observations allows us to use a variety of observation representations which are combined to form the abstractions learned by MMU. This provides flexibility in the choice of observation representation which is lacking in existing methods for the online learning of state abstractions.